Media usage monitoring and measurement system and method

ABSTRACT

Media monitoring and measurement systems and methods are disclosed. Some embodiments of the present invention provide a media measurement system and method that utilizes audience data to enhance content identifications. Some embodiments analyze media player log data to enhance content identification. Other embodiments of the present invention analyze sample sequence data to enhance content identifications. Other embodiments analyze sequence data to enhance content identification and/or to establish channel identification. Yet other embodiments provide a system and method in which sample construction and selection parameters are adjusted based upon identification results. Yet other embodiments provide a method in which play-altering activity of an audience member is deduced from content offset values of identifications corresponding to captured samples. Yet other embodiments provide a monitoring and measurement system in which a media monitoring device is adapted to receive a wireless or non-wireless audio signal from a media player, the audio signal also being received wirelessly by headphones of a user of the monitoring device.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a divisional of U.S. patent application Ser. No. 14/175,811,filed Feb. 7, 2014, which is a continuation of U.S. patent applicationSer. No. 13/914,870, filed Jun. 11, 2013, which is a continuation ofU.S. patent application Ser. No. 12/954,604, filed Nov. 24, 2010, whichis a divisional of U.S. patent application Ser. No. 11/139,330, filedMay 26, 2005, which claims priority to U.S. Provisional PatentApplication No. 60/574,836, entitled, “Open-ended device-independentmedia usage monitoring and measurement system,” filed May 27, 2004. Thecontents of these applications is incorporated herein by reference.

BACKGROUND

Traditional media measurement systems have focused on directlymonitoring channels being utilized by audience members. However, asmedia consumption patterns have become more complex, channel-centricmedia measurement is inadequate for many purposes. It may be desirableto track usage of particular media content independent of channel.Furthermore, although “channel” identification in a traditional mediameasurement system may sometimes be limited to radio or televisionbroadcast station, it is increasingly desirable to track usage of mediaacross several types of media delivery vehicles including radio,television, CD, DVD, computer download, portable media players (e.g. MP3players, iPod), and other vehicles. Furthermore, with respect totracking consumption of advertisements, it may be inadequate to simplytrack channel tuning, because, for example, an audience member may mutea broadcast during commercial periods. Thus simply identifying abroadcast channel does not adequately track whether the audience memberlistened to a particular advertisement.

Some media measurement systems have used codes to “tag” and trackparticular content. However, such systems are limited in that they canonly track content that has been properly encoded.

With the development of more robust content recognition technologies,some content recognition systems have recently been deployed which donot rely on codes. For example, Philips, Shazam Entertainment, andothers have marketed systems for identifying songs played into a mobilephone. Although such systems can be efficiently deployed in the contextof song recognition, deploying such systems in the context of mediameasurement systems poses particular challenges. Continuous searchingagainst a large database of media content can be computationallyintensive. Furthermore, such systems, while increasingly robust, stillreturn some erroneous results, particularly in high-noise environments.

At the same time, the media measurement context provides opportunitiesto utilize data exogenous to a particular audio or video data sample.Such opportunities have thus far been insufficiently exploited for thepurpose of efficiently applying existing content recognitiontechnologies in the media measurement context. Thus, an improved mediameasurement system and method is needed.

SUMMARY

Some embodiments of the present invention provide a media measurementsystem and method that enhances recognition (e.g. in terms of accuracyor efficiency) of the content of a media sample by analyzing informationexogenous to the sample. Some embodiments of the present inventionprovide a media measurement system and method that utilizes audiencedata to enhance content identifications. Some embodiments analyze mediaplayer log data to enhance content identification. Other embodiments ofthe present invention analyze sample sequence data to enhance contentidentifications. Other embodiments analyze sequence data to enhancecontent identification and/or to establish channel identification. Yetother embodiments provide a system and method in which sampleconstruction and selection parameters are adjusted based uponidentification results. Yet other embodiments provide a method in whichplay-altering activity of an audience member is deduced from contentoffset values of identifications corresponding to captured samples. Yetother embodiments provide a monitoring and measurement system in which amedia monitoring device is adapted to receive a wireless or non-wirelessaudio signal from a media player, the audio signal also being receivedwirelessly by headphones of a user of the monitoring device.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several aspects ofparticular embodiments of the invention are described by reference tothe following figures.

FIG. 1 illustrates an exemplary media usage monitoring and measurementsystem in accordance with aspects of an embodiment of the presentinvention.

FIG. 2 illustrates a media measurement method in accordance with aspectsof an embodiment of the present invention.

FIG. 3 illustrates a process for using and generating information suchas that illustrated in FIG. 4 and FIG. 5 and FIG. 3 illustrates anembodiment consistent with aspects of the present invention.

FIG. 4 illustrates a raw play stream generated by a process such as step207 of FIG. 2 ; a clean play stream and clean play list generated by ascrubbing step such as step 209 of FIG. 2 or steps 301, 302, or 305 ofFIG. 3 ; and channel data associated with two channels.

FIG. 5 illustrates a clean play stream, clean play list, channel data,and a clean play list showing deduced play-altering actions. Theillustrated data may be generated by systems and methods in accordancewith an embodiment of the present invention such as, for example, system100 of FIG. 1 method 200 of FIG. 2 , and method 300 of FIG. 3 .

FIG. 6 illustrates timeline structure for data samples generated bysystem and methods such as module 121 of FIG. 1 and steps 201 and 206 ofFIG. 2 .

FIG. 7 shows an example of a computer system that may be used to executeinstruction code contained in a computer program product, the computerprogram product being in accordance with an embodiment of the presentinvention.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled inthe art to make and use the invention, and is provided in the context ofparticular applications and their requirements. Various modifications tothe exemplary embodiments will be readily apparent to those skilled inthe art, and the generic principles defined herein may be applied toother embodiments and applications without departing from the spirit andscope of the invention. Thus, the present invention is not intended tobe limited to the embodiments shown, but is to be accorded the widestscope consistent with the principles and features disclosed herein.

FIG. 1 illustrates an exemplary media usage monitoring and measurementsystem 1000. System 1000 includes media measurement system 100 andmonitoring devices 101 in accordance with aspects of an embodiment ofthe present invention.

As illustrated, monitoring device 101 a includes a microphone 105, amedia player port 102, a headphone port 103, and a data upload port 104.A monitored audience member wears monitor 101 a during a monitoringperiod. When the audience member is consuming media from media player170 a without the use of headphones 160 a (i.e. if media player 170 ahas a speaker), monitor 101 captures audio energy acoustically throughmicrophone 105. However, when the audience member desires to useheadphones 160 a to receive media content from media player 170 a (e.g.an MP3 player, iPod, CD player, DVD player, television, radio, computer,or other media player) he or she can plug headphones 160 a intoheadphone port 103 and plug media player 170 a into media player port102. In that case, monitor 101 a captures audio energy through mediaplayer port 102. Microphone 105 includes a microphone and associatedmicrophone port. However, in alternatives, a monitor may simply have amicrophone port into which a microphone may be plugged or linked viawireless connection. Thus the term “microphone port” will herein referto any electronics capable of receiving energy captured through amicrophone, whether or not a microphone is built into the monitoringdevice itself.

As another example, headphones 160 b are adapted to receive a contentsignal 171 via wireless transmission 171 from media player 170 b.Monitor 101 b is adapted to receive the same content signal 171 frommedia player 170 b. In some embodiments, a syncing process allowsmonitor 101 b to adapt along with headphones 160 b and media player 170b to changes in signal 171 so that monitor 101 b continues to receivethe same signal as headphones 160 b. In some embodiments, syncing may beaccomplished via a wireless and/or automatic process, such as providinga monitor that responsively changes tuning based on tuning changesbetween the media player and the headphones.

Audio data captured by monitors 101, along with associated monitor data(e.g. device ID) (collectively, “audio data/monitor data” 144) may beuploaded to system 100 through upload port 104, directly, via memorydevice transfer (e.g. flash memory card, floppy disk, CD, etc.) orthrough a network such as network 150. The audio data uploaded from amonitor 101 may be raw audio data, or may be audio data that hasundergone one of a variety of levels of processing prior to upload. Forexample, the uploaded audio data may include parameters useable tocalculate audio signatures (sometimes referred to as audio fingerprints)and landmarks or other values useful in the content recognition process.Alternatively, the audio data may comprise pre-calculated signatures,landmarks, or other data useful for content recognition. On the otherhand, in some embodiments, the uploaded audio data may simply representa raw signal corresponding to audio energy received at monitor 101.

System 100 is adapted to receive some audience data 143 externally. Forexample, audience data collected by systems and associated monitorsother than those of system 1000 may be utilized by system 100; databased on externally collected and/or externally analyzed demographic,psychographic, or other audience related data might also be received aspart of external audience data portion 143. Other audience data may begenerated internally by audience data processing module 123 alone or incombination with measurement analysis module 129 and/or other systemcomponents.

Also, system 100 is adapted to receive media player log data 146. Mediaplayer log data 146 may be generated by a media player such as mediaplayer 170 a or 170 b and sent to system 100 through direct connection,memory device transfer or via network 150. Also, a third party mightcollect such log data which then may be made available through directconnection, memory device transfer, or via network 150. Media player logdata 146 includes data logging content played on the media player fromwhich the log data was generated. Although log data 146 does notnecessarily reflect what content was actually heard, to the extent themedia player generating the log data is used primarily by the audiencemember associated with the corresponding monitor 101, it can provide auseful basis for initial testing of captured content as will bedescribed further below.

System 100 is also adapted to receive known content 141 and content metadata 142. Content 141 may include media files associated with a knownpiece of media content (e.g. a song, movie, commercial, television show,video game, etc.). However, it will be understood by those skilled inthe art that “content” in the sense of the known content 141 received bysystem 100 may, in some embodiments, simply refer to data about signalsrepresenting the content and does not necessarily refer to a storedversion of the media content itself; e.g. “content” may be storedsignatures and landmarks derivable from a song's audio, but notnecessarily an audio file useable to play the song itself. On the otherhand, in other embodiments, a system such as system 100 may be adaptedto receive a useable audio file and then derive data from that file suchas signatures, fingerprints, landmarks, etc. that may be readilysearched during the content recognition process. Content meta data 142includes various data relating to content 141. For example, a portion ofcontent 141 might include a particular song as recorded by a particularartist. Corresponding content meta data 142 for that song might includethe various albums or other collections in which that song appears;various radio broadcasts (including station and broadcast times) playingthe song; movies, advertisements or video games including the song;other songs that have sampled the song; and various other relatedinformation. In the context of a television program, meta data alsomight include, for example, segment identifications and correspondingtime lengths. By distinguishing content data from content meta data inthis manner, it is possible to store content data for a single contentID only once, and then add various pieces of meta data as appropriate.For example, a TV program, or program segment, might initially have metadata relating to particular broadcasts. However, at a later date, metadata might be added relating to, for example, a DVD version of theprogram, without needing to store the program content itself more thanonce.

Turning to the details of system 100, audio data and monitor data (e.g.device ID) is received at sample control processing module 121. Module121 provides monitor data, portions of which might be audience data, toprocessing module 123. Module 121 also divides the audio data intosamples and determines which samples should be submitted for furthercontent recognition testing. In alternative embodiments, the audio datamay already have been divided into samples prior to receipt by system100 (e.g. at monitor 101). In such alternatives, a module such as module121 might either use the provided divisions or combine the data and thenre-divide it into samples as determined by the system. Module 121 may beused either to submit a sample to initial test and filtered searchmodule 122 or to content recognition system 124.

Initial test and filtered search module 122 conducts one or more initialidentification attempts. Several different types of initialidentification attempts might be performed. Module 122 may test acaptured sample against content associated with the most recentlyidentified content. As another possible initial test, module 122 maytest the sample against content identified from a media player log file.Another initial identification test may be carried out in a variety ofways. For example, filtered search module 122 might use audience datafrom audience data processing module 123 to form parameters and thenpass those parameters to content recognition system 124 which then usesthe parameters to conduct a filtered search of its content.Alternatively, module 122 might be adapted to utilize an instance ofcontent recognition system 124 that includes content pre-selected basedupon relevant audience data. Some examples of relevant audience datathat may be used either to form search parameters or to constructtargeted content include:

-   -   age, gender, and other demographic information about the        audience member (using the corresponding monitoring device)        along with media consumption patterns of other persons sharing        similar demographic characteristics    -   media consumption patterns of the audience member based on        questionnaire input or based on past content identifications        associated with that audience member    -   media consumption patterns of other audience members who have        consumed other media content that has also previously been        consumed by the audience member using the corresponding        monitoring device    -   location of the audience member (alone and/or relative to other        audience members)        Of course, the above are just some examples of audience data        that may be utilized by system 100.

After completing the initial identification attempt, filtered searchmodule 122 passes the identification result to ID result module 125. Ifan identification has been made, the result is passed to play streamgenerator 126 and to sample control processing module 121. If noidentification has been made, then content recognition system 124performs a search using a larger portion of its content to identify thedata sample. The result is passed to ID result module 125 which in turnpasses the result to play stream generator 126 and to sample controlmodule 121. Content recognition system 124 is adapted to implement oneor more known content recognition methods that identify content byextracting parameters from a media signal and applying an algorithm tothose parameters to search for a content match. Those skilled in the artwill recognize that many such methods and algorithms exist. One suchexample is described in U.S. patent application Ser. No. 09/839,476entitled “System and Methods for Recognizing Sound and Music Signals inHigh Noise and Distortion” by Wang et al. and published Jun. 27, 2002with publication number US2002/0083060. Aspects of other such examplesare described in the following published PCT applications by KoninklijkePhilips Electronics N.V.: “Fingerprinting Multimedia Contents”(WO2004/044820, published May 27, 2004), “Fingerprint Extraction”(WO2004/030341 published Apr. 8, 2004), and “Improvements in andRelating to Fingerprint Searching” (WO2004/040475 published May 13,2004). Such distinct methods and algorithms may have varying degrees ofefficiency, accuracy, and applicability to certain types of content (ordifferent settings of a particular algorithm might have differentefficiency depending on content type). Thus, in some embodiments, asystem such as system 100 may intelligently select among multiple suchalgorithms (or multiple settings of a single algorithm) based onexpected content characteristics. For example, algorithm A might befaster and more accurate at identifying silence than algorithm B, andthus a system such as system 100 of FIG. 1 having a correspondingsubsystem for content recognition (such as system 124) might achieveenhanced performance by submitting a sample for processing by algorithmA rather than algorithm B if recent identification results indicatesilence. Other differences corresponding to content types (e.g. speakingaudio versus music audio) might also be exploited based on prior contentidentification results to select a most efficient recognition algorithmfrom a plurality of algorithms. In such an alternative, if thealgorithms required different audio data parameters, a monitor mightsend two sets of parameters for each sample corresponding to eachalgorithm and then the system would select between the two sets asneeded. Alternatively, the two sets of parameters might be calculated bythe system based on raw audio data received from a monitoring device.

Play stream generator 126 generates a raw play stream relating asequence of samples to content identification results (for an example ofa portion of a raw play stream, see FIG. 3 and accompanying text). Theraw play stream is then processed by play stream scrubber 127 whichanalyzes the sequence of sample identification results utilizingsequence data processing module 128. Based on analysis of sequence dataand, where appropriate, analysis of audience data via interaction withaudience data module 123, play stream scrubber 127 generates a cleanplay stream and clean play list (see FIG. 3 and accompanying text) whichcan then be used by measurement analysis module 129. Results of themedia measurement analysis can be obtained via report generator 130.

In alternative embodiments, many of the system components andcorresponding functions performed within a system such as system 100might instead be implemented on the monitoring devices such asmonitoring device 101. For example, many of the functions performed bysample control processing module 121 might instead be performed at amonitoring device. To cite but one example in more detail, if amonitoring device maintains an ongoing reciprocal connection to obtainsample identification results of a media measurement system, then themonitoring device can select which samples to send to the system forrecognition analysis. The monitoring device might then also adjustparameters such as sample resolution, the length of a sample timewindow, and the recognition algorithm selected. This is just one exampleof how the illustrated embodiment might be modified to shift systemcomponents from a central system to a monitoring device. Many other suchvariations will be apparent to those skilled in the art.

Also, it will be understood that although FIG. 1 illustrates aparticular exemplary division and relationship between “modules,” thedivision illustrated may be readily modified without departing from thespirit and scope of the present invention. For example, in variousimplementations, the illustrated modules may be combined into largermodules or the functions performed may be distributed across severalmodules. The term “module” and the associated illustrated division ofsystem components is chosen for purposes of ease of description only anddoes not limit how particular systems consistent with the presentinvention might be constructed.

FIG. 2 illustrates a media measurement method 200 in accordance withaspects of an embodiment of the present invention. One or more elementsof method 200 may be carried out, for example, by system 100 of FIG. 1or other similar systems. At step 201, method 200 receives audio datafrom a monitoring device. Step 201 caches a series of samples anddetermines the sample resolution with which samples are submitted forcontent identification. In particular, as described further below (seeFIG. 4 and accompanying text), sample resolution refers to how manysamples in a given sequence of samples are processed for contentidentification. Step 201 also determines length, or “time window” ofeach sample.

At step 210, characteristics of the audio data are analyzed to attemptto identify the source of transmission or storage of the media played byan individual based on audio characteristics of that signal. Forexample, the process attempts to recognize particular types of encodingor compression, or identifies sound associated with screen refresh on atelevision or computer monitor, or recognizes compression or frequencyrange of FM or AM radio, or CD or DVD. If the storage or transmissionmedium is identified, the results of step 210 can be added to a playlist and/or utilized in step 211 to identify channel (see below forfurther description of step 211).

At step 202, identification of a sample in a series of samples isattempted via initial test methods. In particular, step 202 determineswhether the audio data in the tested sample matches data associated withthe content ID corresponding to the most recently identified sample.Step 202 may also or instead test the sample against content IDsobtained using data from a media player log file associated with theaudience member using the corresponding media monitor. Step 203determines whether successful identification occurred at step 202. Ifyes, then the identification result is provided to step 206 and 207. Ifno, then step 204 searches for a content match against targeted contentselected based at least in part upon additional audience data. Step 205determines whether step 204 obtained an identification. If yes, then theidentification result is provided to steps 206 and 207. If no, then step208 searches for a content match against a larger portion of thecontent. With respect to steps 202, 204, and 208, there are two possibleresults: an identification has been made or no identification has beenmade. No identification at steps 202 and 204 leads to the furtheridentification attempt at step 208. Whatever the result is at step 208(identification or no identification) that result is passed to steps 207and 206.

At step 206, an identification result is used to adjust sampling controland selection parameters for one or more subsequent samples. Parametersthat might be adjusted include resolution, backtrack selection, sampletime window, or recognition algorithm selected. For example, if a samplehas been identified successfully, and the identification indicates amatch with the prior sample, the sampling resolution might be decreasedfrom ⅕ to 1/10, meaning that, in a sequence of samples, nine samplesrather than four are skipped before selecting the next sample to analyzefor identification. As another example, if an identification result doesnot match the result of the prior sample identified, then step 206 mightinitiate a back track analysis and select an intervening sample that hasnot yet been analyzed. Step 201 would then pass the sample selected frombacktracking to step 202 for attempted identification. As anotherexample, resolution might be increased when recent identificationresults suggest an extended period of silence or absence of knowncontent.

As yet another example, once media content for a limited series ofsamples has been identified, the time window size for each sample mightbe decreased from, for example, 5 seconds to 3 seconds. This would meanthat samples submitted for attempted identification would be shorter inlength. This can create efficiencies because once the content isidentified, the process needs only to detect a change in content, andthis can, in some cases, be accomplished utilizing a smaller amount ofaudio data, thereby preserving system resources.

Adjustment of sampling parameters is described further below in thecontext of FIG. 5 and accompanying text.

Continuing with the description of FIG. 2 , step 207 generates a rawplay stream indicating a series of samples and correspondingidentification results. Step 209 scrubs the raw play stream based uponsample sequence data (described further below in the context of FIGS.3-4 and accompanying text) and based upon audience data. The resultingclean play stream may be converted to a play list listing the contentcaptured and indicating the order in which it was captured. Step 209attempts to identify the channel by searching known channel data to findan apparent content sequence match to the play list. If the storage ortransmission medium of the captured sample has been identified at step210, that information may be made available for step 211 and allows step211 to use that information to filter its search of channel data (e.g.,only search channel data including CD channels, or only including radiobroadcast channels, etc.). If step 211 successfully identifies thechannel, information such as content sequence data for the identifiedchannel may be used by step 212 to further scrub or rescrub the playstream, particularly with respect to any apparently incorrect contentidentifications that could not be corrected based upon sample sequencedata as utilized in step 209.

FIG. 3 illustrates a process 300 for using and generating informationsuch as that illustrated in FIG. 4 and FIG. 5 . Some of the functionsaccomplished by process 300 are similar to functions accomplished bysteps 207, 209, 211, and 212 of FIG. 2 . Process 300 illustrates a moredetailed example of an embodiment consistent with aspects of the presentinvention.

Referring to process 300, depending on available information, either aplay stream is made available to step 301 or a play list is madeavailable to step 302. Step 301 prepares a play stream for furtherprocessing necessary for channel matching by scrubbing (i.e. reviewingfor missing or incorrect data) the play stream using non-channel data.Step 302 prepares a play list for further processing necessary forchannel matching by scrubbing (i.e. reviewing for missing or incorrectdata) the play list using non-channel data. With respect to steps 301and 302, the non-channel data selected might be selected particularlyfor the purpose of better preparing the play stream or play list for achannel matching process. Step 303 receives either a raw (i.e. “notscrubbed”) play stream or play list directly or receives a scrubbed playstream or play list from step 301 or 302. If necessary, step 303converts received data to a format useful in matching to a channel (e.g.converts a play stream to a play list where a play list format is usefulfor channel matching, or converts a play stream or play list to amathematical representation that is useful for channel matching).

Step 304 attempts to match the play stream or play list (orcorresponding data format representing the play stream or play list)against known channel data to identify a channel (e.g., an album,samples of an album presented for marketing—e.g. on a web page—, radiobroadcast, television broadcast, theater version of a movie, DVD versionof a movie, etc.) associated with the elements of the play list or playstream. Step 306 determines whether a channel match was achieved. Ifyes, then step 305 uses channel data and non-channel data to scrub orfurther scrub the play stream or play list for purposes of creating aclean play list. If the play list or play stream has already beenscrubbed with non-channel data in either step 301 and 302, then the playstream or play list may not need to be further scrubbed with non-channeldata at step 305. In that case, step 305 just uses channel data tofurther scrub the play stream or play list. However, it is possible thatscrubbing the play list or play stream with non-channel data forpurposes of creating data useful for channel matching will be somewhatdifferent than doing so for purposes of creating a clean play list, thusnon-channel data is referenced again in the illustration of claim 305.

If step 306 determines that a channel match was not achieved at step304, then step 307 scrubs the play stream or play list with non-channeldata to create a clean play list. Step 309 determines whether includingdeduced user actions on the clean play list is desired. If yes, thenstep 308 creates a clean play list including deduced user actions. Ifno, then step 310 creates the clean play list without including deduceduser actions.

FIG. 4 illustrates Raw Play Stream 410 generated by a process such asstep 207 of FIG. 2 ; Clean Play Stream 450 and Clean Play List 490generated by a scrubbing step such as step 209 of FIG. 2 or steps 301,302, or 305 of FIG. 3 ; Channel Data 470 and Channel Data 490. Referringto FIG. 4 , Raw Play Stream 410 itemizes a sequence of analyzed samplesand relates that sequence to corresponding identification results in thecontent ID column. The content ID result indicates either a referencecorresponding to a particular piece of content or indicates that noidentification was made. The example in FIG. 4 assumes that no userplay-altering activities took place and this example does not rely oncontent offset data in scrubbing the raw play stream to obtain a cleanplay list. However, note that in other examples, such as that of FIG. 5, sample sequence data (“ssd”) includes content offset data. Forpurposes of presenting and analyzing sample sequence data as describedherein, the times in list 410 may be relative to an arbitrary value(e.g. “to”) rather than representing an absolute time; however, in manyinstances, a log time will in fact indicate the actual time the samplestarted to be captured.

Sample sequence data may be used to “scrub” Raw Play Stream 410. Byanalyzing the sequence of data on Raw Play Stream, it is possible tolocate and correct any identifications that are likely to be incorrector to supply missing identifications where the system has not otherwisebeen able to provide an identification.

Sample sequence data can be used to derive portions of Clean Play Stream450 from raw play list 410 in the following manner: Referring to RawPlay Stream 410, the content IDs associated with samples 1, 3, 4, 6, and7 all indicate the same media content referenced as “song51.” Thecontent ID value associate with sample 2 indicates that no content hasyet been successfully associated with that sample and the content IDvalue associated with sample 5 indicates “movie81.” Given this patternof sample sequence data, it is reasonable to assume that the personusing the corresponding monitor listened to song51 through the timeperiod associated with the audio data of samples 1-7, and thus, the“scrubbed” version of the play stream, i.e. Clean Play Stream 450,indicates song51 for each of samples 1-7.

Information about the sequence of content in a play stream can be usedto generate a play list and identify a channel, and the identifiedchannel data can also potentially be used to help further “scrub” a playstream. For example, the sequence of consumed content represented onPlay Stream 450 is searched against known channel data (e.g. particularCD albums, broadcasts, or other particular storage or transmission mediacarrying a particular sequence of content). In this example, simplifiedfor illustrative purposes, known channel data includes channel data 470,associated with channel ID “radio 35,” and channel data 480, associatedwith channel ID “album12.” Referring to channel data 470, each rowrelates a content ID to a start time and end time. In this example,start and end times are referenced with respect to an arbitrary starttime value “C(0).” For some channels (e.g. CDs, DVDs, video games) onlysuch arbitrary reference times will be available. For other channels(e.g. broadcasts, concerts, etc.), absolute time might be both availableand useful. However, for the purpose of the analysis illustrated in thisexample, times that are relative only and not necessarily absolute aresufficient.

With respect to further “scrubbing” the Raw Play Stream 410, in thisexample, channel data may be used to correct the apparentmisidentification of sample 29. Note that the sample sequence data alonein this example does not provide a clear basis for correcting sample 29.Sample 28 is identified as “ad49,” sample 29 as “song81,” and sample 30as “song 35.” However, given the sequence “song21, ad49, and song35” isassociated with a known sequence of the radio broadcast referenced as“radio 35,” there is a basis in channel data 470 for believing that theproper content ID for sample 29 is “song35” rather than “song81.” Thus,Clean Play Stream 450 indicates a change relative to Raw Play Stream 410with respect to the content ID for sample 29. Note that a clean playstream and clean play list are not necessarily generated sequentially.For example, to the extent a channel identification and associatedchannel data are necessary to “scrub” a raw play stream, portions of aclean play list may be determined as part of obtaining data necessary tomake channel selection. In the present example, “radio 35” is identifiedbased on a particular content sequence, and that information helpspopulate Clean Play Stream 450 with a corrected content ID value forsample 29 relative to the sample 29 content ID value on Raw Play Stream410.

Audience data may also be used in the “scrubbing” process. For example,if “song81” is radically different from the user's known consumptionhabits and “song35” is within the user's known consumption habits,another basis might be provided for suspecting that “song35” rather than“song81” is the correct content ID for sample 29 of Play Stream 410.

It will be understood by those skilled in the art that in the context ofdata sets that might, in alternative examples, make up a “raw playstream,” “a clean play stream,” or a “clean play list,” the particularorganization, division, and content of the data may vary considerablyfrom that illustrated in FIG. 4 (as well as that illustrated in FIG. 5below) without departing from the spirit and scope of the presentinvention. To cite but one example, such play streams and/or play listsmight include several data fields in addition to those illustrated.Moreover, the illustrated data sets might not be collected in the sametable but, depending upon database architecture, might be spread out indifferent tables and/or relational structures. Furthermore, the datasets similar to those illustrated might not be stored at all in adatabase but rather might be generated on the fly and then discarded asmere intermediate steps to generating a particular report that thesystem is asked to produce. These and other variations will be readilyapparent to those skilled in the art. The examples shown are chosenprimarily for usefulness in illustrating aspects of a particularembodiment. While they are useful in illustrating certain underlyingprinciples of certain aspects of the present invention, they should notbe considered, in and of themselves, to limit the scope of theinvention.

FIG. 5 illustrates Clean Play Stream 510, Clean Play List 560, ChannelData 570, and Clean Play List 590 showing deduced play-altering actions.The illustrated data may be generated by systems and methods inaccordance with an embodiment of the present invention such as, forexample, system 100 of FIG. 1 method 200 of FIG. 2 , and method 300 ofFIG. 3 . Aside from the traditional time shifting activity of, forexample, recording a television broadcast and playing it at a latertime, various play-altering activities may be undertaken by an audiencemember. For example a media player may be paused, reversed, fastforwarded, and some media players have modes in which skipping back orforward in played content can occur almost instantly, including withinthe smallest time division useful for a particular data set (e.g., lessthan five seconds, less than one second, etc.). Generally, theseactivities of manipulating the pace and order in which media is consumedwill be referenced as audience play-altering actions or activities.These activities may be deduced, for example, during scrubbing step 209of FIG. 2 , or, as illustrated more expressly, during step 308 of FIG. 3.

Continuing with the description of FIG. 5 , Play Stream 510 presents aseries of log times, content IDs, and content offsets. In theillustrated example, times given in the log time column begins with toand ends with t₀+19. Each log time indicates a time at which capture ofthe analyzed sample began, where to simply represents a beginning timefor samples on Play Stream 510. Each content ID indicates an identifierof the content that has been identified as corresponding to the capturedsample. In the example illustrated, “TV program 27” refers to a TVprogram segment and “TV program 28” refers to another program segment ofthe same program. “Ad23” and “ad25,” on the other hand, refer toseparate advertisements. Each content offset value represents a timeoffset from the beginning of a particular piece of content. For example,row 8 of Play Stream 510 corresponds to a sample whose beginning islocated 9 time units from the beginning of TV program 27, thus thecontent offset is referenced as (0)+9.

Play List 560 contains a row for each internally continuous mediaconsumption event. The start log time and end log time for each suchevent are listed in each row. If an entire piece of content (e.g. a TVprogram segment, or an entire TV ad) is listened to withoutinterruption, that would correspond to a row of Play List 560. On theother hand, if the data from Play Stream 510 suggests a discontinuitywithin consumption of a particular piece of content, suchdiscontinuities form boundaries between consumption events that formrows of List 560. The meaning of these concepts may be clarified bydiscussing further details of the illustrated example.

Discontinuities in consumption of the identified media may be identifiedby comparing the progression of sample log times to progression ofcorresponding content offset times in Play Stream 510. For example,referring to rows 7 and 8 of play stream 510, log times progress fromt₀+6 to t₀+7; however, the corresponding content offsets progress, withrespect to the start of TV program 27, from (0)+3 to (0)+9. Had themedia consumption from one sample to the next been continuous, one wouldexpect that the second offset value (in row 8) would have been (0)+4rather than (0)+9. Therefore, a boundary between media consumptionevents can be deduced. Thus, rows 3 and 4 of List 560 indicate twoconsumption events with respect to the same piece of media content (TVprogram 27). By contrast, row 1 and row 2 of Play Stream 510 are, onPlay List 560, collapsed into a single row (row 1) because the contentoffset progression from row 1 to row 2 of Play Stream 510 suggestscontinuous consumption of the same piece of media content.

Referring to Play List 560, the “end log time” of one event also definesthe “start log time” of the next event. In other examples, particularlyif continuous time is not assumed, a play list might be constructedusing start log times that are different than the end log time of theprevious consumption event.

Referring to Clean Play List With Deduced Actions 590, it can be seenthat a play list can be constructed that supplements the information inPlay List 560 with information about the apparent actions of theaudience member. Rows 2, 4, 6, 8, and 11 of List 590 include suchinformation about deduced actions.

To the extent such play-altering actions occur within the same piece ofcontent, audience member actions may be deduced without content sequencedata for a particular channel. For example, referring to rows 2, 3, 4,5, and 6, of Play Stream 510, the content offset values of rows 2 and 6indicate advancing for one time unit within TV program 27 while nocontent ID (or corresponding offset value) is available for rows 3-5.From this information, and the fact that log times of rows 3-5 togethercorrespond to progression through three time units, it is reasonable todeduce that the audience member paused a media player for three timeunits from log time t₀+2 to log time t₀+5. That deduced action isrecorded in row 2 of Play List 590.

On the other hand, in other contexts, it may be necessary to refer tocontent sequence data for a particular channel to be sufficientlyconfident in deducing the details of an audience member's play-alteringactivity. For example, referring to Play Stream 510, data in rows 15-17indicate one or more consumption discontinuities in which aplay-altering action appears to have crossed a content boundary. Row 15indicates consumption of the beginning of ad23; row 16 indicates noidentified content, and row 17 indicates consumption of ad24 from apoint two time units after the beginning of that ad. From thisinformation alone, it is difficult or impossible to determine how muchplay-altering activity occurred and whether any content pieces wereskipped altogether. However, content sequence data for an identifiedchannel may be used to supplement the information gap. Data 570 listscontent sequence data for a channel identified as “TV C4.” “TV C4” maybe identified during a method portion such as step 211 of process 200 ofFIG. 2 as previously described. Channel data may be searched thatincludes the content identified in Play Stream 510 or Play List 560. Achannel whose content sequence matches or closely matches content onsuch a stream or play list may be identified as the channel thatdelivered the consumed content. In this case, Play Stream 510 and PlayList 560 include content associated with the following IDs: TV program27, ad23, ad25, and TV program 28. Because channel data for “TV C4”indicates a sequence of: TV program 27, ad23, ad24, ad25, and TV program28, for purposes of this example, it can reasonably be assumed that themedia depicted on Stream 510, List 560, and List 590 was originallydelivered to the relevant media player by channel “TV C4.” In real worldcontexts, however, it is quite possible that more data would have to beobtained (e.g. a matching more entries on a play list entries against alonger series of content sequence data of a channel) in order to bereasonably confident that the channel has been correctly identified. Theamount of data matched in the illustrated example has been chosenprimarily for ease of illustration.

Sequence Data 570 for channel TV C4 indicates that ad23 as broadcastlasted three time units (see row 2), ad25 as broadcast lasted three timeunits (see row 4), and between ad23 and ad25, another ad, ad24, wasbroadcast and lasted three time units. From this information, and theinformation in either Play Stream 510 or Play list 560, it can bededuced that the audience member heard the very beginning of ad23, butthen fast forwarded seven time units (during the span of 1 time unitbased on the log time data) to just past the beginning of ad25. Thisdeduced action is recorded in row 11 of List 590.

It should be noted that, to the extent actions take less than a singletime unit, rows may be added to List 590 relative to List 560 to recordsuch actions. For example, referring to row 4 of List 590, because nolog time elapsed between the end of the event in row 3 and the beginningof the event in row 5, row 4 has been added showing the “skip forward 5units.” That the user “skipped” forward rather than “fast” forwarded canbe deduced from the fact that while there is no content ID gap betweenrows 3 and 4 of List 560, there is a gap in the content offset times forthose rows. Thus, though the log times and content ID alone suggestcontinuous consumption, the gap in offset times suggests a “skip”forward that took the same or less time than the length of a single timeunit.

Those skilled in the art will recognize that the method aspects justdescribed in the context of generating and using Stream 510, List 560,Data 570, and List With Deduced Actions 590 reflect just one example ofhow play-altering actions might be deduced using systems and methodsconsistent with those of the present invention. To cite but one example,rather than explicitly generate a play list, mathematicalrepresentations of play stream data and content sequence data may beused to identify discontinuities and deduce play-altering activities. Tocite but one other example, play-altering might be deduced from apre-generated play list including content offset data without needing toanalyzing play stream data from which the play list was generated.

FIG. 6 illustrates timeline structure for data samples generated bysystem and methods such as module 121 of FIG. 1 and steps 201 and 206 ofFIG. 2 .

As illustrated, sample set 600 reflects a division of audio data intosixteen samples, three of which are selected for analysis. The samplesselected for recognition analysis are referenced generally as the “n₀^(th),” n₁ ^(th),” and “n₂ ^(th)” samples. Samples between the selectedsamples are not separately numbered, but are marked off between verticallines along the horizontal length of timelines structure 600 asillustrated. Each sample has a sample time window length “i.”Furthermore, the sample set has varying “resolutions” including r₀=⅕between the n₀ ^(th) and n₁ ^(th) samples and rt= 1/10 between the n₁^(th) and the n₂ ^(th) samples. In other words, “resolution” here refersto the portion of samples being selected during a period of time. In thepresent example, a process portion such as steps 201 and 206 of process200 of FIG. 2 has, based on the identification results of the n₀ ^(th)and n₁ ^(th) samples, adjusted the resolution from ⅕ to 1/10. This couldoccur, for example, as previously described, because the n₀ ^(th) and n₁^(th) samples match the same content ID and process 200 determines thatthe likelihood that the next selected sample using a resolution of ⅕would also match justifies decreasing the resolution to 1/10 so that thenext sample selected for analysis is the n₂ ^(th). However, if the n₂^(th) sample does not match the n₁ ^(th) sample, then process 200 canmake a backtrack decision and go back in the series of samples toselect, for example, the (n₁+6)^(th) sample for content recognitionanalysis in order to pinpoint more closely where in the play stream thecontent change occurred. The order in which samples are selected foranalysis in this example is illustrated by arrows 1, 2, and 3 in FIG. 6.

Although not illustrated in FIG. 6 , as previously described, steps 201and 206 of method 200 can also determine based on identification resultsthe time window length can be adjusted. In the context of FIG. 6 , thiswould correspond to changing the value of “i,” the time window length ofa particular sample.

In alternative embodiments to what is illustrated and described in FIGS.1-6 , a record is created of individual media usage over time by meansof an open-ended measurement system that employs portable and stationarydetection devices to generate digital fingerprints of the video signalportion of the media content itself and identifies media played in theenvironment by means of matching such fingerprints against a database ofknown media. Such a video-based system may be built upon directelectrical connections and optical sensors (rather than microphones).Thus, various alternatives to the exemplary methods discussed in thecontext of the audio data-based embodiments illustrated in FIGS. 1-6rely instead on video data.

FIG. 7 shows an example of a computer system 700 that may be used toexecute instruction code contained in a computer program product 760 inaccordance with an embodiment of the present invention. Computer programproduct 760 comprises executable code in an electronically readablemedium that may instruct one or more computers such as computer system700 to perform processing that implements the system 100 of FIG. 1and/or accomplishes the exemplary method 200 of FIG. 2 . Theelectronically readable medium may be any medium that either stores orcarries electronic signals (including signals referred to as electricalsignals and signals referred to as electromagnetic signals) and may beaccessed locally or remotely, for example via a network connection. Theexecutable instruction code in an electronically readable medium directsthe illustrated computer system 700 to carry out various exemplary tasksdescribed herein. The executable code for directing the carrying out oftasks described herein would be typically realized in software. However,it will be appreciated by those skilled in the art, that computers mightutilize code realized in hardware to perform many or all of theidentified tasks without departing from the present invention. Thoseskilled in the art will understand that many variations on executablecode may be found that implement exemplary methods within the spirit andthe scope of the present invention.

The code or a copy of the code contained in computer program product 760may be stored in memory 710 for execution by processor 720. Computersystem 700 also includes I/O subsystem 730 and peripheral devices 740.I/O subsystem 730, peripheral devices 740, processor 720, and memory 710are coupled via bus 750.

Those skilled in the art will appreciate computer system 700 illustratesjust one example of a system in which a computer program product inaccordance with an embodiment of the present invention may beimplemented. To cite but one example of an alternative embodiment,execution of instructions contained in a computer program product inaccordance with an embodiment of the present invention may bedistributed over multiple computers, such as, for example, over thecomputers of a distributed computing network.

Although particular embodiments have been described in detail andcertain variants have been noted, various other modifications to theembodiments described herein may be made without departing from thespirit and scope of the present invention. Thus, the invention islimited only by the appended claims.

1. A media measurement method comprising: using one or more computers togenerate a play stream of content identification results correspondingto a sequence of submitted data samples, the submitted data samplescorresponding to video data samples captured at a monitoring deviceassociated with a monitored audience member; and using one or morecomputers to utilize sample sequence data and content offset data todeduce play-altering actions of the monitored audience member, thesample sequence data corresponding to a sequence of the contentidentification results, the content offset data including contentoffsets associated with the content identification results, a contentoffset indicating a time position of a content identification resultwithin identified content.
 2. The method of claim 1, wherein: to utilizecomprises, for a particular plurality of submitted data samples andcorresponding content identification results, comparing a progression oflog times associated with capture timing of each of the particularplurality of submitted data samples and a progression of content offsetsof the corresponding content identification results.
 3. The method ofclaim 2 further comprising: generating a play list including contentoffset data as part of deducing media play-altering actions of amonitored audience member.
 4. The method of claim 2 further comprising:utilizing sample sequence data together with corresponding channel datato deduce play-altering actions of a monitored audience member.
 5. Themedia measurement method of claim 2 wherein a clean play list isgenerated including deduced play-altering actions.
 6. The mediameasurement method of claim 2 wherein the play stream is a raw playstream.
 7. The media measurement method of claim 2 wherein the playstream is a clean play stream.
 8. The method of claim 2 wherein toutilize further comprises determining if one or more discontinuitiesexist in the progression of content offsets.
 9. The method of claim 2wherein the content offset data is derived from a comparison conductedby a computerized content identification system comparing the submitteddata samples against known content such that content offset data isobtainable without using time codes accompanying the captured video datasamples.
 10. A media measurement method comprising: using one or morecomputers to generate a play stream of content identification resultscorresponding to a sequence of submitted data samples, the submitteddata samples corresponding to video data samples captured at amonitoring device associated with a monitored audience member; and usingone or more computers to utilize sample sequence data and content offsetdata to deduce play-altering actions of the monitored audience member,the sample sequence data corresponding to a sequence of the contentidentification results, wherein play-altering actions include actionsthat alter at least one of a pace and a sequence of a media stream, thecontent offset data including content offsets associated with thecontent identification results, a content offset indicating a timeposition of a content identification result within identified content,wherein to utilize comprises, for a particular plurality of submitteddata samples and corresponding content identification results, comparinga progression of log times associated with capture timing of each of theparticular plurality of submitted data samples and a progression ofcontent offsets of the corresponding content identification results. 11.The method of claim 10 wherein play-altering actions include at leastone of fast forwarding, rewinding, and pausing a media stream.
 12. Themethod of claim 10 wherein play-altering actions include at least one ofskipping forward and skipping backward in a media stream.